Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 405: model specific priors #565

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Conversation

SamuelBrand1
Copy link
Collaborator

@SamuelBrand1 SamuelBrand1 commented Dec 19, 2024

This PR closes #405.

This PR adds a remake_latent_model function which is placed into the pipeline and either dispatches a pass through of the default latent model, or has a specified choice which reacts to information on mcmc diagnostics from our last full run (but not to scores).

Following f2f with @seabbs the general pattern has been to reduce a priori variance especially for difference processes and ExpGrowthRate models. The exception after some prior predictive checking (no comparison to data) is DirectInfection models which now have higher correlations when in an AR mode. This makes sense upon reflection since the DirectInfection models the residuals away from inferred log-It level.

See next comment for prior predictive sampling:

Copy link
Contributor

Try this Pull Request!

Open Julia and type:

import Pkg
Pkg.activate(temp=true)
Pkg.add(url="https://github.com/CDCgov/Rt-without-renewal", rev="specific-priors", subdir="EpiAware")
using EpiAware

@SamuelBrand1
Copy link
Collaborator Author

priorpred_gi_mean=2 0_gi_std=2 0_igp=EpiAware EpiInfModels DirectInfections_latent_model=ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=2 0_gi_std=2 0_igp=EpiAware EpiInfModels DirectInfections_latent_model=diff_ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=2 0_gi_std=2 0_igp=EpiAware EpiInfModels DirectInfections_latent_model=rw_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=2 0_gi_std=2 0_igp=EpiAware EpiInfModels ExpGrowthRate_latent_model=ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=2 0_gi_std=2 0_igp=EpiAware EpiInfModels ExpGrowthRate_latent_model=diff_ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=2 0_gi_std=2 0_igp=EpiAware EpiInfModels ExpGrowthRate_latent_model=rw_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=2 0_gi_std=2 0_igp=EpiAware EpiInfModels Renewal_latent_model=ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=2 0_gi_std=2 0_igp=EpiAware EpiInfModels Renewal_latent_model=diff_ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=2 0_gi_std=2 0_igp=EpiAware EpiInfModels Renewal_latent_model=rw_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=10 0_gi_std=2 0_igp=EpiAware EpiInfModels Renewal_latent_model=ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=10 0_gi_std=2 0_igp=EpiAware EpiInfModels Renewal_latent_model=diff_ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=10 0_gi_std=2 0_igp=EpiAware EpiInfModels Renewal_latent_model=rw_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=20 0_gi_std=2 0_igp=EpiAware EpiInfModels Renewal_latent_model=ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=20 0_gi_std=2 0_igp=EpiAware EpiInfModels Renewal_latent_model=diff_ar_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70
priorpred_gi_mean=20 0_gi_std=2 0_igp=EpiAware EpiInfModels Renewal_latent_model=rw_priorpredictive=true_scenario=SmoothOutbreakPipeline_tspan=1_70

@SamuelBrand1 SamuelBrand1 requested a review from seabbs December 19, 2024 21:41
@codecov-commenter
Copy link

codecov-commenter commented Dec 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.96%. Comparing base (0d4095e) to head (af6dfb6).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #565   +/-   ##
=======================================
  Coverage   90.96%   90.96%           
=======================================
  Files          60       60           
  Lines         863      863           
=======================================
  Hits          785      785           
  Misses         78       78           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@seabbs seabbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I haven't reviewed the priors in detail. Will do so in the morning. The implantation seems correct but um yeah a bit horrifying.

Did you consider using double dispatch on remake latent model by infection model and latent model then using accessors to update the latent model?

@SamuelBrand1
Copy link
Collaborator Author

Sorry I haven't reviewed the priors in detail. Will do so in the morning. The implantation seems correct but um yeah a bit horrifying.

I agree, the basic problem was that we identified:

  • Diff AR is quite a different process to AR (obviously) because, approximately, its variance growth like $t$.
  • ExpGrowthRate is quite different to other IGPs because its cumsum on the log-scale.
  • DirectInfections on log-scale is therefore one less diff version of ExpGrowthRate.

Once I started reasoning on that you quite quickly end up with this horrible special case prior implementation. I couldn't think of a smooth dispatch, although maybe I just need to think a bit more.

Tidying this:

  • Treating the default as one good choice at least reduces the bloat.
  • Reasoning a bit more on how we expect the variance to be a priori and then reversing back towards a good choice of prior dists on the parameters would be much smoother.

Copy link
Collaborator

@seabbs seabbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've looked again at the code and it follows what we discussed. At a minimum there needs to be information in the doc string about the specific choices that are hardcoded. I would like to see an issue to update this implementation as its very brute force and hard to reason across.

seabbs
seabbs previously approved these changes Dec 20, 2024
Copy link
Collaborator

@seabbs seabbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SamuelBrand1 the prior predictive plotting is super super useful here and I like how they are looking across models. I also like the use of a time scale for the non-stationary processes. I am happy to go forward with these choices with my only required action being documenting the logic near the hard coding

@SamuelBrand1
Copy link
Collaborator Author

SamuelBrand1 commented Dec 20, 2024

Really good feedback.

Ongoing, I'm writing more logic into the doc strings :

This function sets the target standard deviation for an infection generating process (igp)
based on whether the latent process representation of its dynamics are stationary or non-stationary.

Stationary Processes

For Renewal process log(R_t) in the long run a fluctuation of 0.75 (e.g. ~ 75% of the mean) is not unexpected.
For Exponential Growth Rate process r_t in the long run a fluctuation of 0.1 is not unexpected e.g. going from
rt = 0.1 (7 day doubling time) to rt = -0.1 (7 day halving time) is a 2 x 0.1 fluctuation.
For Direct Infections process log(I_t) in the long run a fluctuation of 1.0 (i.e an order of magnitude) is not unexpected.

Non-Stationary Processes

For Renewal process log(R_t) in a single time srep run a fluctuation of 0.025 (e.g. ~ 2.5% of the mean) is not unexpected.
For Exponential Growth Rate process r_t in a single time step run a fluctuation of 0.005 is not unexpected.
For Direct Infections process log(I_t) in a single time step run a fluctuation of 0.025 is not unexpected.

@SamuelBrand1
Copy link
Collaborator Author

And I'm not convinced the hard code version matches the scientific reasoning. Just doing something to check that before any commits

@SamuelBrand1
Copy link
Collaborator Author

And I'm not convinced the hard code version matches the scientific reasoning. Just doing something to check that before any commits

To address this I've got #566 to merge into this branch. I think that this makes the logic much clearer, and therefore implements what we want to implement. Albeit it needs scientific review.

Copy link
Collaborator

@seabbs seabbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After merging in the changes this is good to go

@seabbs seabbs enabled auto-merge December 23, 2024 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Model specific priors capability
3 participants